Exploring Satisfiability: Instance Families and Experimental Design

Experimental evaluation of SAT algorithms has been practiced and debated with considerable intensity over the last decade. Typical comparisons involve a collection of unrelated benchmark instances. Algorithms are evaluated on the basis of their total or average execution time for the collection. We propose a significant departure from this approach. By introducing equivalence classes of closely related instances for each reference formula, we can experimentally deduce the 95% confidence interval of the mean time-to-solve and other significantly correlated metrics. Instances derived from the reference formula that are perceived ‘hard’ by the algorithm under test may exhibit max/min ratios of a metric that can range anywhere from 2 to 1000 and beyond. In such cases, comparisons based on single formulas have no statistical merit. A total of four class types are formalized and experiments are performed on at least 32 instances in each class. We introduce an experimental design environment and present experimental results that reveal startling, and statistically significant, differentiations between three state-of-the art algorithms and a vanilla DPLL algorithm. As a side benefit, these results provide a number of insights to improve each of these algorithms. Note.This document has been published for viewing on the Web in PDF and Postscript formats. “Permission to make digital/hard copy of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage, the copyright notice, the title of the publication and its date appear, and notice is given that copying is by permission of CBL. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.” c ©2001 authors and CBL, All Rights Reserved Citation. If you choose to cite this report, please add the following entry to your bibliography database: @techreport{2001-TR@CBL-02-Brglez, author = "F. Brglez and X.Y. Li and M.F. Stallmann", title = "{Exploring Satisfiability: Instance Families and Experimental Design}", institution = "{CBL, Computer Science Dept., NCSU, Raleigh, NC 27695}", number = "2001-TR@CBL-02-Brglez", month = "November", year = "2001", note = "{Also available at {\tt http://www.cbl.ncsu.edu/\-publications/\-\#2001-TR@CBL-02-Brglez}}", } Technical Report 2001-TR@CBL-02-Brglez (November 2001) Exploring Satisfiability: Instance Families and Experimental Design ∗ Franc Brglez Dept. of Computer Science NC State University Raleigh, NC 27695, USA [email protected] Xiao Yu Li Dept. of Computer Science NC State University Raleigh, NC 27695, USA [email protected] Matthias F. Stallmann Dept. of Computer Science NC State University Raleigh, NC 27695, USA Matt [email protected]